Moving Object Segmentation: All You Need Is SAM (and Flow)

🌈 Abstract

The objective of this paper is motion segmentation – discovering and segmenting the moving objects in a video. The authors investigate two models for combining the Segment Anything Model (SAM) with optical flow to harness the segmentation power of SAM with the ability of flow to discover and group moving objects. The first model, FlowI-SAM, adapts SAM to take optical flow as input. The second model, FlowP-SAM, takes RGB as input and uses flow as a segmentation prompt. These simple methods outperform previous approaches by a considerable margin in both single and multi-object benchmarks. The authors also extend these frame-level segmentations to sequence-level segmentations that maintain object identity, again outperforming previous methods.

🙋 Q&A

[01] Moving Object Segmentation

1. What are the two models proposed in the paper for combining SAM with optical flow? The two models proposed are:

• FlowI-SAM: Adapts SAM to take optical flow as input
• FlowP-SAM: Takes RGB as input and uses flow as a segmentation prompt

2. How do these models leverage the strengths of SAM and optical flow? FlowI-SAM leverages the ability of SAM to accurately segment moving objects against the static background, by exploiting the distinct textures and clear boundaries present in optical flow fields. FlowP-SAM effectively leverages the ability of SAM on RGB image segmentation, with flow information acting as a selector of moving objects/regions within a frame.

3. How do the authors extend the frame-level segmentations to sequence-level? The authors introduce a matching module that auto-regressively chooses whether to select a new object or propagate the old one based on temporal consistency, in order to maintain object identities throughout the video sequence.

[02] Experiments

1. What are the key datasets used for evaluation? The authors evaluate on single-object benchmarks (DAVIS2016, SegTrackv2, FBMS-59, MoCA) and multi-object benchmarks (DAVIS2017, DAVIS2017-motion, YouTube-VOS2018-motion).

2. How do the authors' methods perform compared to previous approaches? The authors' methods, both FlowI-SAM and FlowP-SAM, outperform all previous approaches by a considerable margin on both single and multi-object benchmarks, at both frame and sequence levels.

3. What are the key findings from the ablation studies? The ablation studies show that:

• Using optical flow with multiple frame gaps improves performance by capturing more consistent motion information
• Combining flow features by averaging performs better than taking the maximum
• Injecting flow prompts into the SAM architecture and using moving object scores as guidance yields significant improvements

</output_format>